Move output from `CSV` files to a `SQL` db #403

Grufoony · 2026-02-06T11:52:53Z

Closes Add possibility to connect to a sql db #379

TODO:

Save edges and nodes data in the db + find a way to identify networks.
Rewrite webapp to extract data from the db

examples/slow_charge_rb.cpp

  dynamics.setSpeedFluctuationSTD(0.1);

+  // Connect database for saving data
+  dynamics.connectDataBase(OUT_FOLDER + "simulation_data.db");


examples/slow_charge_tl.cpp

    dynamics.setDataUpdatePeriod(30);  // Store data every 30 time steps

+  // Connect database for saving data
+  dynamics.connectDataBase(OUT_FOLDER + "simulation_data.db");


src/dsf/base/Dynamics.hpp

+#ifdef __APPLE__
+    std::time_t const t = std::chrono::system_clock::to_time_t(now);
+    std::ostringstream oss;
+    oss << std::put_time(std::localtime(&t), "%Y%m%d%H%M%S");


test/mobility/Test_dynamics.cpp

-          std::filesystem::remove(testFile);
+          SQLite::Statement cols(db,
+                                 "SELECT street_id, density, avg_speed FROM road_data");
+          while (cols.executeStep()) {


test/mobility/Test_dynamics.cpp

-              std::ifstream file(entry.path());
-              REQUIRE(file.is_open());
+        // Evolve until agent reaches destination (with limit)
+        for (int iter = 0; iter < 1000 && dynamics.nAgents() > 0; ++iter) {


test/mobility/Test_dynamics.cpp

-              break;
-            }
+          SQLite::Statement cols(db, "SELECT distance_m, travel_time_s FROM travel_data");
+          while (cols.executeStep()) {


test/mobility/Test_dynamics.cpp

+          SQLite::Statement cols(db, "SELECT distance_m, travel_time_s FROM travel_data");
+          while (cols.executeStep()) {
+            auto distance = cols.getColumn(0).getDouble();
+            auto time = cols.getColumn(1).getDouble();


test/mobility/Test_dynamics.cpp

+            auto distance = cols.getColumn(0).getDouble();
+            auto time = cols.getColumn(1).getDouble();
+            CHECK(distance > 0.0);
+            CHECK(time > 0.0);


test/mobility/Test_dynamics.cpp

-
-          file.close();
-          std::filesystem::remove(testFile);
+        for (int iter = 0; iter < 1000 && dynamics.nAgents() > 0; ++iter) {


codecov · 2026-02-06T12:18:28Z

Codecov Report

❌ Patch coverage is 89.32039% with 33 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.42%. Comparing base (9611d9e) to head (c48f2a9).
⚠️ Report is 21 commits behind head on main.

Files with missing lines	Patch %	Lines
src/dsf/mobility/FirstOrderDynamics.cpp	72.09%	12 Missing ⚠️
src/dsf/mobility/RoadDynamics.hpp	93.67%	11 Missing ⚠️
test/mobility/Test_dynamics.cpp	87.17%	0 Missing and 10 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #403      +/-   ##
==========================================
+ Coverage   85.81%   87.42%   +1.61%     
==========================================
  Files          54       53       -1     
  Lines        6190     5941     -249     
  Branches      671      658      -13     
==========================================
- Hits         5312     5194     -118     
+ Misses        862      728     -134     
- Partials       16       19       +3

Flag	Coverage Δ
unittests	`87.42% <89.32%> (+1.61%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

src/dsf/mdt/PointsCluster.hpp

+    /// @brief Copy assignment operator for PointsCluster.
+    /// @param other The PointsCluster to copy from.
+    /// @return Reference to this PointsCluster.
+    PointsCluster& operator=(PointsCluster const& other) = default;


src/dsf/mdt/TrajectoryCollection.cpp

         cluster_radius_km,
-         max_speed_kph,
-         min_duration_min](auto& pair) {
+         max_speed_kph](auto& pair) {


github-advanced-security

Cppcheck (reported by Codacy) found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

webapp/styles.css

  }

+  /* Database modal styles */
+  .modal {


webapp/styles.css

+    display: none;
+  }
+
+  .modal-content {


webapp/styles.css

+    text-align: center;
+  }
+
+  .modal-content h2 {


webapp/styles.css

+    margin-bottom: 20px;
+  }
+
+  .db-input-group input[type="file"] {


webapp/styles.css

+    border-color: #4CAF50;
+  }
+
+  .db-status {


webapp/styles.css

+    text-align: center;
+  }
+
+  .modal-content h2 {


webapp/styles.css

+    border: 2px dashed #ccc;
+    border-radius: 5px;
+    width: 100%;
+    box-sizing: border-box;


webapp/styles.css

+    min-height: 20px;
+  }
+
+  .db-status.error {


webapp/styles.css

+    color: #c62828;
+  }
+
+  .db-status.success {


webapp/styles.css

+    color: #2e7d32;
+  }
+
+  .db-status.loading {


github-advanced-security

Stylelint (reported by Codacy) found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

…PCHECK

webapp/script.js


+// Grufoony - 9/2/2026
+// TODO: make this dynamic based on data range
+const MAX_DENSITY = 200;


webapp/script.js

+// Grufoony - 9/2/2026
+// TODO: make this dynamic based on data range
+const MAX_DENSITY = 200;
+const MAX_DENSITY_INVERTED = 1 / MAX_DENSITY;


Copilot

Pull request overview

This PR migrates simulation output from CSV files to a SQLite database, updates the C++ simulation engine to write structured results into that DB, and rewrites the webapp to load and visualize simulations directly from the SQLite file (including selecting which simulation run to view).

Changes:

Add SQLite-backed persistence for per-street data, aggregate stats, travel data, simulation metadata, and network topology (edges/nodes).
Update the webapp UI/JS to load a user-selected SQLite database in-browser (sql.js) and select a simulation to visualize.
Remove the legacy AdjacencyMatrix API, tests, and Python bindings that depended on it; refresh examples/tests to use DB saving.

Reviewed changes

Copilot reviewed 22 out of 23 changed files in this pull request and generated 10 comments.

Show a summary per file

File	Description
webapp/index.html	Adds sql.js dependency and a modal to select/load a SQLite DB.
webapp/styles.css	Styles for the new DB-load modal.
webapp/script.js	Switches data loading from CSV/config to SQLite queries + simulation selection flow.
src/dsf/base/Dynamics.hpp	Adds simulation `id` and SQLite DB connection handling.
src/dsf/mobility/RoadDynamics.hpp	Adds DB schema initialization, network dump, and DB-backed saving in `evolve()`.
src/dsf/mobility/FirstOrderDynamics.hpp / .cpp	Implements dumping simulation parameters into a `simulations` table.
src/dsf/mobility/RoadNetwork.hpp / .cpp	Removes `AdjacencyMatrix`-based construction and related includes/ctors.
src/dsf/base/Network.hpp	Removes `AdjacencyMatrix`-based constructor.
src/dsf/bindings.cpp	Drops `AdjacencyMatrix` bindings and replaces CSV save APIs with `saveData()` + DB connect exposure.
CMakeLists.txt	Fetches/links SQLiteCpp and exposes it via `dsf` target.
examples/slow_charge_tl.cpp / slow_charge_rb.cpp	Switch examples from CSV saving to DB saving via `connectDataBase()` + `saveData()`.
test/mobility/Test_dynamics.cpp	Replaces CSV-based save tests with DB table/schema assertions using SQLiteCpp.
test/mobility/Test_graph.cpp	Removes `AdjacencyMatrix` constructor subcase usage.
test/base/Test_AdjacencyMatrix.cpp	Deletes `AdjacencyMatrix` test suite.
src/dsf/base/AdjacencyMatrix.hpp / .cpp	Deletes `AdjacencyMatrix` implementation.
src/dsf/dsf.hpp	Removes `AdjacencyMatrix` include.
src/dsf/mdt/TrajectoryCollection.cpp	Fixes lambda capture list (removes unused capture).
src/dsf/mdt/PointsCluster.hpp	Adds copy assignment operator.
.gitignore	Ignores `*.ipynb`.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/dsf/mobility/RoadDynamics.hpp

webapp/script.js

src/dsf/mobility/RoadDynamics.hpp

src/dsf/base/Dynamics.hpp

src/dsf/mobility/FirstOrderDynamics.cpp

Copilot · 2026-02-09T10:44:55Z

webapp/script.js

+  for (const ts of timestamps) {
+    // Get all road_data for this timestamp and simulation
+    const dataResult = db.exec(
+      `SELECT street_id, density FROM road_data WHERE simulation_id = ${selectedSimulationId} AND datetime = '${ts}'`


road_data is created by the simulator with a density_vpk column (see C++ schema), but this query selects density. This will return NULL/throw and the map will show no densities. Update the query (and downstream usage) to read density_vpk (and optionally also pull avg_speed_kph, counts, etc. if needed).

Suggested change

`SELECT street_id, density FROM road_data WHERE simulation_id = ${selectedSimulationId} AND datetime = '${ts}'`

`SELECT street_id, density_vpk FROM road_data WHERE simulation_id = ${selectedSimulationId} AND datetime = '${ts}'`

webapp/script.js

github-advanced-security bot found potential problems Feb 6, 2026

View reviewed changes

Grufoony force-pushed the db branch 2 times, most recently from 5157ae1 to 808fcc9 Compare February 6, 2026 15:54

github-advanced-security bot found potential problems Feb 6, 2026

View reviewed changes

Grufoony and others added 14 commits February 9, 2026 11:16

Init

37afe86

Exclude Jupyter Notebooks

89e1259

Replace CSV output with SQL [WIP]

ae00fec

Remove streetMeanSpeed functions

7ae721f

One save function to rule them all

0a3389f

Add simulation id to keep using the same database

9ebdfdd

MacOS things...

6e24360

Fix clang compilation warnings

412bbcc

Remove unused and depprecated \'AdjacencyMatrix` class

f320e0b

Save edge and node tables on db

2293f63

If on Windows, disable SQLITECPP_RUN_CPPCHECK to avoid issues with CP…

9a6f98f

…PCHECK

Update webapp to use db

0054efa

Correct density scale

9733e38

Name refactoring and docs

8961d75

Grufoony force-pushed the db branch from f01e859 to 8961d75 Compare February 9, 2026 10:36

Grufoony requested a review from Copilot February 9, 2026 10:36

Copilot started reviewing on behalf of Grufoony February 9, 2026 10:39 View session

github-advanced-security bot found potential problems Feb 9, 2026

View reviewed changes

Copilot AI reviewed Feb 9, 2026

View reviewed changes

Grufoony added 3 commits February 9, 2026 12:41

Fix binding workflow

1bb6ed8

Copilot suggestions

f8fc1bf

Add SQLiteCpp

e48d960

Version 5.0.0

58a15aa

Grufoony force-pushed the db branch from e792b6c to 58a15aa Compare February 9, 2026 12:10

Grufoony added 2 commits February 9, 2026 13:51

Remove unused file

8055302

Fix Windows wf

c48f2a9

Grufoony force-pushed the db branch from 0a65e53 to c48f2a9 Compare February 9, 2026 13:26

Grufoony merged commit 45e99fb into main Feb 9, 2026
44 checks passed

Grufoony deleted the db branch February 9, 2026 13:52

	`SELECT street_id, density FROM road_data WHERE simulation_id = ${selectedSimulationId} AND datetime = '${ts}'`
	`SELECT street_id, density_vpk FROM road_data WHERE simulation_id = ${selectedSimulationId} AND datetime = '${ts}'`

Move output from CSV files to a SQL db #403

Move output from CSV files to a SQL db #403

Uh oh!

Conversation

Grufoony commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

Check notice

codecov bot commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Check notice

Check notice

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Check warning

Check warning

Check warning

Check warning

Check warning

Check notice

Check notice

Check notice

Check notice

Check notice

github-advanced-security bot left a comment

Choose a reason for hiding this comment

Uh oh!

Check notice

Check notice

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI Feb 9, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Move output from `CSV` files to a `SQL` db #403

Move output from `CSV` files to a `SQL` db #403

Grufoony commented Feb 6, 2026 •

edited

Loading

codecov bot commented Feb 6, 2026 •

edited

Loading